X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C9142D.1BE2C73B@onstor-exch02.onstor.net>; Thu, 11 Sep 2008 09:40:33 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Drastic performance problems when connected to Cisco 6500 switches?
Date: Thu, 11 Sep 2008 09:40:31 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0B968FBB@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0B8A5C23@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Drastic performance problems when connected to Cisco 6500 switches?
Thread-Index: AckIeZ6JWTf4goP+SOWw/5jK02ppaAKVlEawAC+n2BA=
References: <BB375AF679D4A34E9CA8DFA650E2B04E0B6A9044@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E0B8A5C23@onstor-exch02.onstor.net>
From: "Jean-Paul Junod" <jeanpaul.junod@onstor.com>
To: "Rich LaReau" <rich.lareau@onstor.com>,
	"Carl Ciengi" <carl.ciengi@onstor.com>,
	"Caeli Collins" <caeli.collins@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>
Cc: "dl-cstech" <dl-cstech@onstor.com>

I was recently able to get the details of Savvis's issue with their 6500
switch.  They were able to pin the connectivity problem down to the 6148
line cards that were in the switch.  Besides the specific problem of
packet loss when connected to Onstor ports, they said that these line
cards were not designed for an enterprise environment and could only
handle a combined total throughput of 1gbps (not per port).  They
replaced the 6148s with 6748s.  This change not only resolved the
connectivity issue but also tripled the performance that they were
seeing overall.

J.P.=20

-----Original Message-----
From: Rich LaReau=20
Sent: Tuesday, September 09, 2008 4:06 PM
To: Carl Ciengi; Caeli Collins; Brian Montero; Jean-Paul Junod; Fay
Chong
Cc: dl-cstech
Subject: Drastic performance problems when connected to Cisco 6500
switches?


Hi all,

I think we need to get this type of information captured, documented and
ideally tested locally and understood.  Do we think that the issue with
Savvis was essentially the same problem that Carl found at OCC?  What
about other mysterious performance sites-- might they all be related?

Rich


Jean-Paul's Savvis update:

Since Savvis has not given us the full report of the network
reconfiguration that they implemented to fixed their Onstor affecting
performance issue; I have written up the following problem summary, as
far as I'm aware of it.=20

Originally the problem was thought to just be related to certain vlans
and ips when they were bound to particular Onstor FP ports. The problem
manifested as a big drop in performance that was seen on the client
side. When packet traces were taken from the clients at the same time
they were connected to the seemingly bad Onstor ports, the traces showed
lots of packet loss and retransmission. This prompted us to investigate
the network more and led us to discover that the Onstor vsvrs were
retransmitting because they were not receiving acknowledgements from the
client; and that the client was not getting the missing packets at all.=20

In the end after much testing of the network Savvis was able to pinpoint
the problem to the line cards of the network switches. By moving all the
vlans to different switches they not only got rid of the connectivity
issues, but were also able to double the throughput of even the best
non-lossy performance they were seeing before. In order for us to fully
understand why they were seeing the original problem (especially the
port specific issue) We need Savvis's detailed final summary.=20

________________________________

From: Carl Ciengi=20
Sent: Wednesday, August 27, 2008 12:18 PM
To: dl-cstech; dl-se
Cc: Caeli Collins; Rich LaReau; Doug Cook
Subject: Issues at OCC ahve been identified and are not with ONStor


=20

So in further testing with OCC in Chicago, IL we have discovered three
areas of concern with there environment.

First is with a BUS speed problem they have with SUN Sparks in use. They
have a number of systems that have only one port that will support 66mhz
add-in cards and that is being used for HBA's for storage. This forces
them to use a 33mhz slot for Ethernet thus reducing the speed and
performance of those SUN systems.=20

Second is with SUN 490 Sparks, they have two multispeed Ethernet ports
that chokes when running at 1000 but have found that if they run a snoop
at the same time on the port the speed actually increases to normal 1000
performance levels. Another way to improve this is if they slow the
ports down to 100mb vs. 1000 it performs a little better but not what
they could get at 1000.

Third is with CISCO 6500 switches and the multi-port cards installed in
them. It seems that the problem may be with the ACICS chips and the
cards physical configuration. If ether channel is configured and they
use sequential ports on the same card their performance tanks. If you
move the required ports across multiple cards the problem goes away.

=20

Note: I am working on getting the model number and firmware release
information on the switches for internal reference stayed tuned.

=20



Carl Ciengi
Senior Systems Engineer

ONStor, Inc.
office: 630.942.1473
mobile: 630.414.6617

Carl.Ciengi@onstor.com <mailto:Carl.Ciengi@onstor.com>=20
http://www.onstor.com <http://www.onstor.com>=20

=09

=09

=20

